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(54) Process and system for generating a full color image or multispectral image from the image 
data of a CCD image sensor with a mosaic color filter. 



(57) Process and system for producing color 
images and/or multispectral images in which an optical 
arrangement (3) images a scene on a CCD image sen- 
sor (2), said sensor being covered by a mosaic filter with 
a periodically repeating basic pattern (GM). A following 
analog/digital converter (5) produces digital signals 
from all pixels pjj of the CCD image sensor (2). The 
basic pattern (GM) consists of a minimum of five differ- 
ent filter elements. In addition, an image memory (6) is 
provided which is divided into image memory levels (B 0 , 
B 1f ....,B nF ). The process comprises the following steps: 

storing of values (wertp]Dl[0]) of each individual 
pixel (py) of the CCD image sensor (2) in a dedi- 
cated level B 0 of the image memory (6). said values 
being determined by the CCD image sensor (2) and 
converted by a following analog/digital converter (5) 
into digitial values; 

determining an "unfocussed image" in the following 
way: for each color f and for each pixel p the neigh- 
boring pixels of color f are measured, their values 
read from Bq and combined to arrive at a weighted 
average value; 

storing of values (wert[i][j][f]) for all the different filter 
elements on the CCD image sensor in dedicated 

image memory levels (B 1p B2 B nF ); 

calculating final color values (C 1t C2 C nF ) by 

means of additive image sharpening; and 

storing the final color values (C 1t C 2 , C nF ) in 

dedicated image memory levels (B 1( B2 B nF ). 
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The invention relates to a process for generating a full color image or multispectral image from the image data of a 
CCD image sensor with a mosaic color filter, said process having an optical arrangement capable of imaging a scene 

5 on a CCD image sensor which is covered by a mosaic filter with a periodic basic pattern. 

In addition, the invention relates to a system for generating a full color image or a multispectral image from the 
image data of a CCD image sensor with a mosaic color filter, said system having an optical arrangement for imaging a 
scene on a CCD image sensor, said CCD image sensor being covered by a mosaic filter with a periodic basic pattern 
and being provided with an analog/digital converter for delivering digital signals from all pixels of the CCD image sensor. 

w Electronic cameras, equipped with only one matrix image sensor (e.g. a CCD image sensor) for taking images and 

which nevertheless produce a color image with a single exposure, exist both in the form of video cameras (moving pic- 
ture) and electronic digital cameras (still image). A mosaic filter (CFA = color filter array) allows such cameras to differ- 
entiate between colors. Such a filter is placed directly on the surface of the CCD image sensor whilst each cell (pixel) 
of the CCD image sensor is covered by a filter element (pixel filter) having suitably chosen spectral transparency. 

is The document "The Seybold Report on Publishing Systems" (12 th December, 1994) discloses a digital camera 
which can take images with a single exposure of a CCD image sensor. The individual cells of the CCD image sensor 
are covered by four different filter elements and the filter elements are placed directly on the individual cells of the CCD 
image sensor. The four filter elements employed for taking an image are red, green, blue and blue-green. The filter ele- 
ments are arranged on the CCD image sensor in a mathematical pattern. In order to determine the color of each indi- 

20 vidual pixel of the CCD image sensor, the color values are first determined by an algorithm over an area of 64 pixels 
and afterwards each pixel is compared with its nearest neighbor. If this is not sufficient to determine the color, then the 
comparison with the nearest neighbors is extended to a larger area. In transition regions from one color to another, up 
to one thousand calculations are required for one pixel. Since the data are not available in standard RGB format, a con- 
version has to be performed which takes a high performance computer approximately 7 minutes per image to complete. 

25 Video cameras also exist which have combinations of additive and subtractive filter sets, such as, for example, 
green, cyan, magenta, yellow or green, white, cyan, yellow (see EP-A-0 570 204). Such combinations are intended to 
better adjust the spatial scanning frequencies for luminance and chrominance signals to the bandwidths of the video 
system and hence, a reduction of the color Moire effects. However, such combinations do not achieve, nor are they 
aimed at improving color fidelity or improving the spectral selectivity of the image. In the video systems discussed in 

30 EP A 0 570 204 the signal processing is performed in "real time" without intermediate storage of images. Thus, the 
technical set-up is completely different than in the case of digital still image cameras as referred to in the process 
according to the present invention. 

Most electronic color cameras have mosaic filters with three different colors. In almost all electronic cameras these 
different colors are arranged in rows and columns in a periodically repeating pattern. The pattern created by repetition 

35 of a (2.2) matrix with identical diagonal elements is known as a Bayer pattern and is widely used as a mosaic color filter. 
Highly developed processes for color interpolation in mosaic filters with RGB filters in the Bayer pattern (US-A-5,373 
322 and US- A- 5,382,976) exploit the peculiarities of this pattern in a decisive way, that is they use the dominance of 
green and the low incidence of fuzziness in the case of red and blue. In addition, the processes disclosed in US-A- 
5.373,322 and US- A-5, 382,976 have been developed in such a way that regardless of the contents of the image, the 

40 process is able to decide whether more use should be made of the row-wise or of the column-wise correlations of the 
image data. This produces texture-like artifacts in those image areas which have no details, but which are noisy. 

An electronic camera has to be able to produce image data sets in a conventional format, i.e. TIFF or TARGA. This 
means that a value has to be provided for each pixel and for each color belonging to the data set. Thus, for each pixel 
an R-value, a G-value and a B-value have to be available in the RGB data set. It is evident that the raw data of the cam- 

45 era do not satisfy this requirement: they contain only one color value for each pixel, that is for a color which changes 
from pixel to pixel. Hence, the camera must have a means by which it can use the raw data to calculate these missing 
color values. Each process for doing this is called a process for the color interpolation of mosaic filters or, in short, a 
process for color interpolation. 

The intention is to arrive at values which would be obtained when taking images without using a mosaic filter at 

so arbitrarily short time intervals with the different filter elements nF and using a filter wheel which contains the nF filter 
colors as filters covering the overall surface area. 

The quality of such a process is a complicated matter. As the lack of objective information is at best replaceable by 
plausible, but never by completely certain assumptions, it is always possible to construct instances of images whose 
quality would seemingly indicate that the process is a failure. For example, it is easy to see that for each set of raw data 

55 there is a distribution of white light which would generate these raw data. On the other hand, there are often plausible 
reasons why such phenomena need not be taken into consideration. The most noticeable artifacts ("aliasing") occur in 
the case of images in which the image signal differs greatly from pixel to pixel. In most electronic cameras with relatively 
few pixels, a "blurr filter" ensures that no such finely structured light distributions can occur in the area of the sensor. 
Distinctions are made between the following, partly overlapping quality complexes: 
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1 . Faithfulness to detail: the ability to reproduce fine structures which extend only over a few pixels, the reproduction 
being made without greatly changing the form. When comparing two different systems, improved faithfulness to 
detail is evidenced by improved legibility of the text, the elements of which only cover a few pixels. 

5 2. Artifacts: these include all image structures of which, without possessing detailed knowledge of the camera, it 

can be said that they are very probably not the same as the original. Examples are: 

a) color edge artifacts: strongly colored striped pattern in the transition area between evenly, but differently 
colored areas. 

10 b) color Moire: semi-even, usually striped color variations over periodically structured areas. 

c) flat-field artifacts: textures, defects and jumps in intensity along the lines of such image areas which appar- 
ently lie outside the focal area of the optical image and which, thus, do not show any sharp differences in inten- 
sity. 

r5 3. Color fidelity: considered here for monochromatic areas which extend over many pixels and in all directions. 

It is the object of the present invention to avoid the undesired appearance of texture produced by the aforemen- 
tioned processes whilst simultaneously maintaining high image sharpness and suppressing color edge defects. 

A further object of the present invention is to develop and to realize a process in a system so that further develop- 
20 ments are possible such as. for example, the introduction of additional filter colors, without necessitating changes in the 
hardware and software architecture. 

In addition, the process is intended for use in more periodic and pseudo-random distributions of the filters whilst 
maintaining similarly favorable results. In particular, the aim is to improve color fidelity and to enable a conversion of the 
exposure to a type of light with different spectral composition. 
25 In accordance with the present invention this object is attained by means of a process in which a basic pattern is 
built up of at least five different color filter elements and which comprises the following steps: 

storing values of each pixel of the CCD image sensor in a dedicated level of an image memory, said values being 
measured by the CCD image sensor and being transformed by a following analog/digital converter into digital val- 

30 ues; 

deriving an "unfocussed image" so that for each color f and for each pixel p, the neighboring pixels of color f are 
determined, their values read from B Q and combined to yield weighted average values; 

storing the values for all the different filter elements on the CCD image sensor in dedicated image memory levels; 
calculating the final color values by means of additive image sharpening and 
35 - storing the final color values in the dedicated image memory levels. 

In accordance with the present invention a system is also provided in which the basic pattern is built up of at least 
five different filter elements. 

Advantages of the process and the system as disclosed by the present invention consist in that images can be 
40 taken by the CCD image sensor with a single exposure and in that greater color faithfulness is thereby achieved than 
in the case of an RGB image. The pixels of the CCD image sensor are covered by at least five different filter elements. 
According to a preferred embodiment of the invention, the so-called mosaic filter consists of a periodic basic pattern. 

A particularly advantageous aspect of the present invention is the process implemented in the system. To generate 
a full color image, values of the individual pixels on the CCD image sensor are stored in an electronic memory. Using 
45 these data the pixels are determined which are provided with a filter element of a specific color. The color values of the 
individual pixels are determined using the corresponding weighting factors of their nearest neighbors. The final color 
values are determined by means of additive image sharpening. 

In addition, the process and its related system make it possible, when taking a image, to convert the values from 
one illumination situation to another with a different spectral composition, 
so The subject matter of the present invention will now be described with reference to an embodiment shown in the 
drawing wherein: 

Fi 9- 1 represents a schematic view of the system as described in the present invention; 

55 Fl 9 2 shows a top view of a part of the CCD image sensor in which the distribution of the filter elements on 

the surface of the CCD image sensor can be seen; 

Fi 9 3 depicts a top view of a part of the CCD image sensor for which a different distribution of the filter ele- 

ments on the surface of the CCD image sensor has been chosen, and 
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Fig. 4a and 4b show a flow diagram of the process. 

The system according to the present invention, as shown schematically in Fig. 1 , comprises an optical arrangement 
3 which images a scene or an object on a CCD image sensor 2. The CCD image sensor 2 is divided into several pixels 

5 py , i running from 1 to I and indicating the number of columns on the CCD image sensor 2, and j running from 1 to m 
and indicating the number of lines on the CCD image sensor 2. The signals gained from the individual cells of the CCD 
image sensor 2 are conducted via an electrical connection 4 to an analog/digital converter 5. Thereafter, the digital sig- 
nals are stored in an image memory 6. The image memory 6 is divided into several image memory levels B ot B 1 to B nF 
nF representing the number of different filter elements placed on the CCD image sensor 2. The individual image mem- 

w ory levels B 0 , B 1 to B nF are connected to a processor 8 which processes the data required to generate a full color pic- 
ture. In addition, processor 8 is connected to an additional electronic device for the processing and storage of signals. 

If the number of filter elements with different spectral transparency is increased, then it is possible to achieve an 
improvement in color fidelity. A CIE ruling with regard to color values for the light indicent on the CCD image sensor 2. 
requires that the spectral sensitivity of the pixels can be represented as a linear combination of the corresponding func- 

15 tions of the CIE standard observer. This demand is very difficult to meet from a technical point of view and is grossly 
violated in electronic cameras of conventional construction. The result is that no matter how well a system comprising 
camera and display is calibrated, it will not permit a faithful reproduction of color nuances. To be more precise: there will 
always be illuminated objects which appear to the observer as being heterochromatic for which, however, an RGB cam- 
era can only deliver like signals. Regardless of the method of processing the signals, the color reproduction stays the 

20 same: this is in contrast to the observer's visual perception of the original. Naturally, the opposite effect likewise occurs: 
objects visually perceived as being isochromatic are reproduced as being heterochromatic. If a mosaic filter on a CCD 
image sensor contains more than three different filters, then new signals can be produced by forming suitable linear 
combinations of the camera signals: these new signals correspond far more exactly to the visual perception of the orig- 
inal than do the RGB signals. This principle was applied to four filters in the camera model already referred to in the 

25 "Seybold Report on Publishing Systems", 12 th Edition, December 1994. 

In addition, a conversion from one type of illuminating light to another can easily be performed. If it is to be possible 
to draw a reliable conclusion from the color of an illuminated body to its color when illuminated by light having another 
spectral composition, then it is necessary to know the spectral properties both of the reflected light as well as of the illu- 
minating light. Using suitable methods of calculation (spectral reconstruction), the spectral properties can be computed 

30 if the signals are known which the light yields after passing through filters of known spectral transmission. These com- 
putations will be more reliable, the more filter elements there are available for the analysis of the light. Just five suitable 
filter elements instead of the usual three or four, will dramatically improve the reliability of this process. It can be used, 
for example, to take photographs under low-cost halogen light conditions and then to convert to daylight. The spectral 
composition of the illumination can be determined, without using any other devices, by photographing a white reference 

35 body having a known spectral reflection degree under the given illumination or, by always adding this reference body to 
the scene to be photographed. 

In the preferred embodiment of the system according to the present invention, five different filter elements are dis- 
tributed in a specific pattern on CCD image sensor 2 (see Fig. 2). 

In a preferred embodiment, for example, a system is employed comprising of five filter elements whose maximum 

40 spectral transparency has the following wavelengths (Table 1): 

Table 1 
B = 445 nm 
G* = 494 nm 
G = 545 nm 
G"= 570 nm 
R = 595 nm, 

The spectral transparency of the individual filter elements B, G\ G, G" and R being chosen in such a way that it is 
55 possible to achieve the most uniform possible overlapping of the individual wavelength regions. The wavelengths in 
bold-face characters denote a sensor system equivalent to that of the human eye. The wavelengths inbetween are 
termed G* (for 495 nm) and G M (for 570 nm). 

The individual filter elements are arranged on CCD image sensor 2 in a repetitive pattern (see Fig. 2). This repeti- 
tive pattern is used on the entire surface of CCD image sensor 2 (see Table 2). 
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Table 2 

RG B 

B R G 

G" B R 



Fig. 2 shows the arrangement of the individual filter elements on the individual pixels p n Pi 2 . -Pjj of CCD image 
sensor 2, i representing the number of rows and j the number of columns. Only part of CCD image sensor 2 is shown 
here since the pattern given in Table 2 is continued over the entire CCD image sensor 2. Thus, for example, pixel p n 
of the first row and the first column is covered by filter element R. pixel p 12 of the first row and the second column by 
filter element G\ pixel p 13 of the first row and the third column by filter element B. pixel P21 of the second row and the 
first column by filter element B, pixel P22 of the second row and the second column by filter element R, pixel p 23 of the 
second row and the third column by filter element G, pixel p^ of the third row and the first column by filter element G", 
pixel p 32 of the third row and the second column by filter element B and pixel p 33 of the third row and the third column 
by filter element R. This basic pattern GM (Table 2) is continued over the entire CCD image sensor 2: this is best illus- 
trated in Fig. 2. Another possible form of basic pattern GM can be seen in Table 3. Fig. 3 explicitly shows the arrange- 
ment of the individual filter elements with respect to the individual pixels p 12 Py of CCD image sensor 2. 





Table 3 


G" 


G B 


B 


R G 


G 


G' R 



The basic pattern GM as illustrated in Fig. 2 and Fig. 3 are embodiments of multispectral mosaic filters. In addition, 
it is likewise possible to design multispectral mosaic filters which contain IR-sensitive. or even UV-sensitive, filter ele- 
ments. It is expedient in this instance to take the entire spectral range for which the "naked" CCD image sensor 2 is sen- 
sitive, to break it up into transmission bands whilst ensuring the greatest possible uniformity and to provide each band 
with a mosaic type of filter. Such a multispectral mosaic filter camera acts as an imaging spectrometer with high spacial 
resolution and moderate spectral resolution. 

As shown in Fig. 1, the system has a digital image memory 6 with random access (RAM). The image memory 6 
has image memory levels B 0 , B, to B nF which correspond to the number nF of the different filter elements used in basic 
pattern GM. The number of image memory levels Bq. B 1 to B nF is greater by one than the number of different filter ele- 
ments used to build up the basic pattern GM. In the embodiment descrtoed here, exposure of CCD image sensor 2 is 
performed and its pixel values directly transmitted to the first level Bq: access to the pixel values wert[i]0HO] can be made 
for each pixel p^. This provides the starting point for the above- mentioned process. In the course of the process, addi- 
tional nF image memory levels B 1t B;, to B nF are available for the storage of data wertp]0][f], so-called "color-values". 
As mentioned above, nF represents the number of different filter elements. In the embodiments described according to 
the present invention, the number of different filter elements amounts to five. After completion of the process, the image 
memory levels B 1 , to B nF have a "full picture". In addition, the system has a processor 8 which reads data from image 
memory 6 into its own register (not shown) in order to perform calculations with the contents of its own register and to 
write the results into image memory 6. 

Fig. 4a and Fig. 4b show a flow diagram of the process according to the present invention. A scene or image is pho- 
tographed by means of the CCD image sensor 2. In the following step, the values measured by the individual pixels py 
of the CCD image sensor 2 are converted by the analog/digital converter 5 into digital data values wert[i][fl[0]. These 
data are stored for further use in image memory level Bq. In order to obtain a full color image, all pixels p 11( p 12 to p, m 
have to be assigned digital values for each of the nF filter colors. This step is performed using a double loop. The outer 
loop runs via colors f and the inner loop via pixels pjj of the CCD image sensor 2 (see Fig 4a). Those pixels of the CCD 
image sensor 2 which are provided with the filter element of color f and which are to be found in the neighborhood of py 
are determined. This is done by performing an evaluation of the neighboring pixels as will be explained in detail below. 
In addition, the nearest neighbors are given a weighting factor which will depend on the distance to starting pixel pjj. 
These data provide the basis for calculating the values wertpMf] as weighted mean values. Values wert[i]Q][f] are then 
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stored for further use in the dedicated image memory levels B 1 , B 2 to B nF As already mentioned, determination of color, 
determination of the nearest neighbors and determination of the color values of all pixels are steps which are carried 
out for all different filter elements on CCD image sensor 2. Using a suitable process (described below), the final color 
values are determined and likewise stored for further processing in a suitable memory. 

5 As already mentioned, it should be noted that, in order to calculate the full image, all color values C 1 = wert[i][j][1], 

C 2 - wert[i]D][2] to C nF = wert[i][j][f] of each individual pixel p n , p 12 to p, m on the CCD image sensor 2 have to be deter- 
mined for each filter color used. It can be seen from the preferred embodiments shown in Fig. 2 and Fig. 3, that when 
making an exposure, not all pixels of the CCD image sensor 2 provide a signal for each of the filter colors used; hence, 
the color values missing for certain pixels have to be determined. The mosaic filter illustrated in Fig. 2 is taken as an 

w example. Processor 8 reads those pixels from the image memory level B Q which have a value, for example, for the color 
value R. The values for those pixels without a measured value then have to be calculated. According to Fig. 2, a value 

for color value R is available for pixels p 11f P22. P33. Pi4> P25. P36. P17 P41. P52. P63. P44. P55 etc Employing a suitable 

process, color value wert[i][j][R] must be calculated for the missing pixels p 12 , P13. P15. P16 P21. P23. P24. P26--P31- 

P32> P34. P35 P42. P43> P45 etc. The calculation of the values of the other filter colors G, G\ G" and B is carried out 

15 accordingly. 

Processor 8 proceeds as follows to calculate the color values wert[i]D][fl for each pixel py, wherein i = 1 1; j = 

I, m and 

f = 1, nF. 

In a first step the following calculations are made for each color f (numbered 1 ,2 nF) and for each pixel py: 

20 Loading all pixel values wert[i]0][O] by using "evaluation of neighboring pixels" (see description below) wherein the 
pixels belong to color f and whose pixels p rj « are nearest to pixels py, and load them together with their weighting factors 
from the image memory level B 0 . (see description of the evaluation of neighboring pixels). 

Calculating the weighted mean of these data with regard to the aforementioned weighting factors w rj - f . 

Storing of the weighted mean as color value wert[i]fj][f] in the image memory level , B2 B nF belonging to color 

25 t 

In this way a full, but nevertheless intermediate image is obtained and one which is termed here as being an "unfo- 
cussed image". The intention in the construction of this image is as follows: the picture should be free of fine details and 
its color should be reliable in regions in which the color undergoes no rapid change. In order to ensure the absence of 
fine details, the evaluation of neighboring pixels must be performed in such a way that it never defines only one pixel as 

ju being the nearest neighbor. Only in this instance is it possible for the process to generate weighted mean values which 
are not local and which produce the desired reduction in sharpness. It has been shown that a mean value over four or 
five points is ideal. If computing effort and computing time do not pose a problem, it is possible to make the "unfocussed 
picture" more homogeneous by a smoothing filter operation. 

Thereafter, processor 8 calculates the final image according to a process for which two alternative embodiments 

35 are given here: 

first, loading of the following values for each pixel py is performed: 

S = wertBDHOJ 

from image memory 6 and there from image memory level B 0 ; 
40 C 1 = wert[i]Q][1] 

from image memory level B^ 
C 2 = wert[i]Q][2] 

from image memory level B 2 and 



45 C nF = wert[i]G][nF] 

from image memory level B nF 

Determine color f of pixel py according to the method of color evaluation as described in detail in the following. 
The final color values C 1f C 2 ,...C nF are calculated according to the formula (additive image sharpening) 

50 

C k = C k + S -C f for k= 1,...nF 
or the formula (multiplicative image sharpening) 

55 SC k 

C k = — 5fork^1....nF. 



If the result falls below the lowest meaningful digital signal value (as a rule 0), or exceeds the highest meaningful 
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digital signal value (frequently 255), then these limitations are defined as representing the result. In this instance it is 
preferable to reduce the additive correction value S - C f or the multiplicative correction value S/C f to a value which will 
prevent exceeding these limits. 

The final color values C n , C 2 , .C nF are stored in the memory levels B 1t B2,...B nF They are: 

5 

wert[.M1] = C 1 . 
wertlij[fl[2] = Cs 



wert[i][j][nF]=C nF 

10 

With this process it is possible to produce a barely noticeable change (using the additive method) or no change at 
all (using the multiplicative method) in the color of the unfocussed picture and to use the actual signal value S to effect 
luminance adaptation. In the case of the additive method this is achieved by adding white or black, in other words by 
means of color desaturation. In this way. any color edge artifacts become less noticeable: for this reason, the additive 
75 method is generally preferred. In addition, it is simpler to calculate. 

It is to be expected that the process will show little tendency toward color edge artifacts: firstly, because the color 
is calculated by averaging all values, and secondly, because the color is desaturated following each addition of light. 
Since the brightness can be adjusted for each pixel without averaging, good fidelity to detail is also to be expected. 
Experiments have confirmed both expectations. 
20 On the other hand, an artifact is present which is unknown in other processes. Along lines where heavily colored 
areas meet, a colorless pattern is generated which is caused by the geometry of parts of the mosaic filter. It is generally 
not very noticeable. 

The following gives a detailed explanation regarding color determination and evaluation of neighboring pixels for 
the preferred embodiments of the invention. 

25 

Color determination: (determination of the color of the individual pixels of the CCD image sensor) 

In the case of a periodic mosaic f flter (basic pattern GM), as illustrated in Fig. 2 or Fig. 3, the entire mosaic filter on 
the CCD image sensor 2 is built up by periodic repetition of basic pattern GM. The column number of the basic pattern 
30 is termed n Vert and the row number mHor. Thus, nVert = mHor =3 is valid for the pentaspectral pattern proposed in 
accordance with the present invention. The distribution of the colors on the nVer, mHor positions of basic pattern GM 
takes the form of a numerical (nVert, mHor) matrix. Basic pattern GM, for example, has the form shown in Table 3 in the 
case of the pentaspectral mosaic f flter (for the sake of clarification the filter elements were arranged as indicated in Fig. 
3): 

35 

Table 3 
4 3 1 

40 15 3 

3 2 5 



45 The numbers in the matrix are assigned as follows to the individual colors of the filter elements of the basic pattern: 
B == 1 , G* s 2, G s 3, G" = 4 and R = 5. 

On the basis of the matrix shown in Table 3, that pixel p y is determined which has a specific color f = C ^ Color f of 
pixel py is calculated by means of the following equations in which i is the vertical and j is the horizontal position of an 
arbitrary element of the CD image sensor and in which the row-wise and column-wise position of the pixel on the basic 

so patttern GM is represented by \ [okai and j^, respectively. 

i lokaJ : =1 + remainder of the division (i-1)/nVert 
j lokal : =1 + remainder of the division (j-1)/nHor 
C y : = GM [itokajHijoka,] 

55 

In a pseudo-random mosaic filter, color Cy of pixel py is determined for a given i and j as follows: 

r: = 50000 • sin(i) + 30000 • sinQ) 

z1: = biggest integer, smaller or equal to r. 
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r1 : = r - z1 , this number yields 0<r1<1. 

In order to illustrate the significance of this expression, we consider y:=5 • sin(i)+3 • sin(j) as a decimal fraction and 
assume, for the sake of simplicity, that y is positive: 

5 

y = d1, 62 d3 d4 d5 d6 67 d8 

The number r1 can then be developed into a decimal fraction 
0,d6 67 68 

10 r1 is defined by the "higher digits" of y; it is evident that they depend on i and j pseudo-ramdomly. In a last step, r1 is 
adjusted to the desired value range (1,...nF). (nF is the number of the different colors): 

r2: = nF • r1 

z2: = biggest integer, smaller or equal to r2 
75 C u : = 1 + r2-z2. 

This is only one of many procedures which are based on the fact that the higher decimal digits of transcendental 
functions depend pseudo-randomly on the argument of the function, if it varies in big steps compared to the order of 
magnitude which is given by the decimal digit in question. In particuliar, the constants 50 000 and 30 000 can be 

20 replaced by other numbers of a similar order of magnitude, and it is possible to replace sin( ) by cos( ). This purely ran- 
dom distribution of colors has the tendency to generate clusters, that is pixel regions often occur in which one color is 
far more frequently represented than in the case of even distribution. There are likewise surprisingly large areas in 
which one color is completely missing. By varying the process, as can be done in processes for "digital half-toning", 
care can be taken that the color distribution is more uniform, but without giving rise to locally constant patterns, such as 

25 those which are present in periodic mosaic patterns. An obvious method is to take optimizing experiments as a basis 
and to develop such "uniformized random distribution" of the pixel colors over a relatively large field of about 64 • 64 
pixels and then to perform periodic repetition of the pseudo-random pattern. As a result, the advantages of periodic pat- 
terns , i.e. the simpler neighbor algorithm, are also effective in this case. 

30 The evaluation of the neighboring pixels: (determination of the number of pixels py located closest to a particular pixel 
which have the same color as the first pixel). 

As not all pixels py of the CCD image sensor 2 supply the information of a simple color, the missing pixels py have 
to be calculated in order to obtain a color image. The process proposed for making these calculations contains several 
35 parameters to be evaluated which can be freely chosen within certain limits. First of all, the distance between the pixels 
is defined. The distance d of pixel p i1 j 1 to pixel p x2 \2 IS defined as follows: 

d(i1,j1;i2j2): = |i1-»2| + |j1-j2| 

40 or, the geometrical distance between the pixel p^-i and pixel p j2 j2. the latter, however, being less suitable for fast com- 
puting. 

In addition, for each pixel a calculation has to be made to determine the maximum area U(py) to be taken into 
consideration; this fixes an upper limit for the number of "neighboring pixels" and defines the number of computing steps 
required to select these neighboring pixels. This is performed by establishing a number d Max with which it is possible to 
45 calculate the area U(pjj) in question. 

U(P y) = {P rr I I '-''I^Cl Max; I j " i'l Max! 

For each pixel py and each color f the subset of U(p v ) is formed from those pixels whose color f is : 

50 

U(p 0 fl = {p n . 1 1 i-i'lsd Max; I j - r | <d Max ; C y =f} 

For the color occurring the least frequently, the set should also have at least 2 and preferably 4 to 6 elements. This 
can be achieved by selecting a sufficiently large value of d Ma x- Tn'S means, however, that for the color occurring the 
55 most frequently, the set U(Pjj.f) can contain more pixels than are required. In order to take this into consideration, the 
pixels are arranged according to their distance (see above) from the reference pixel Dy and those furthest away are dis- 
carded. 

The following procedure is applied: two pixels from this set are defined as being equivalent when they have the 
same distance from reference pixel p^. In this way, U(pjjf) is divided into equivalence classes. Arranged accorcfing to 
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their increasing distance, these equivalence classes are denoted as follows: 



u 1 (p jj ;0;U 2 ( Pjj ;0; ;U k ( Pij ;f) 

s and the number of their elements are denoted by 

Ui(P§fl.u 2 ( Pjj ;f) u k (p ij: f). 

The above-mentioned possibility of discarding some of these elements is made reality by selecting a natural 
10 number kO with 

1<k0<k 

such that 

15 

HQ 

U p ( Pij ;f)*5 

20 

According to the definition 

NU( Pij fl= U^p-fju U 2 ( Pij ;f)w kj U k0 (p jjf -f) 

25 then yields pixels py of color f which are in the area closest to pixel py. The weight of pixels py now has to be evaluated. 
One possibility is that this weight solely depends on the distance to the reference pixel thus, all pixels of the same equiv- 
alence class have the same weight. This is done by choosing a list g> of kO positive numbers for which the following is 
valid: 

30 co(kO)< <<o (1) 

2>(p) = i. 

p=1 



35 



40 



Hence the weight of a pixel PjjeNU(py;f) is given by 



where the index p is chosen so that 
45 (Py) = U p ( Pjj ;f) 

Thus, the weighted mean value required in the first step of the process is defined as 



50 



55 



wert[i][j][f] = X w(p iT ;f) . wert[i'][niO] <B1 ) 



In this way, the evaluation of the neighboring pixels can be performed each time a color is determined. However, 
the formation of equivalence classes is very computer intensive. In the case of a periodic pattern, computer time can 
be significantly reduced by making preliminary calculations and storing them. In equation (Bl) i and j outside the square 
brackets can be replaced by i loka , and j loka ,, defined in the earlier description of the color evaluation, and i\ j' by i+s, j+t 
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where (s,t) is restricted by corresponding conditions like 



wertp]D][f]= X "(Piiokaijiokaifl ' wertp+s]D+t][0] 

5 (Pir)^NU(P ifokall}of(a :;f) 



with 

10 NU rel (p (j ;f): = {(s,t)| (i+s.j+t) e NU(i ,j;f)} 

With this, for the nVert • nHor • nF values Oiokai JiokahO. * he lists 

NU rel (p jIokaJjk>Kal ;f) = ((s1.t1).(s2,t2) ) 

75 

and 

W (P ilokal jlokal fl = ( W 1 • W 2 ) 

20 have to be calculated, whose length is typically (see above) only 4 to 6. These lists are calculated once as part of the 
software development of the system and stored in a ROM. When this process takes place in a camera with the actual 
values (pjjif) being dependent on (p it0 kai. jiokaifi. the relevant lists are loaded and the weighted mean value simply cal- 
culated as: 

25 wert[i][j][f] = w1 • wertfi+s1]D+t1][0)+ w2 • wert[i+s2][j+t2][0]+.... 

The present invention has been described with reference to a preferred embodiment; however, it is obvious that an 
expert in the field may make modifications in accordance with his capabilities without exceeding the scope of protection 
of the following claims. 

30 

Fig. 4 a 

a. Image recording 

b. Digitization Result : image level B 0 contains photo image data 
35 c. f : = first color 

d. p : first pixel 

e. Evaluate (see evaluation of neighboring pixels) pixels p 1§ ...pk of color f which are in the neighborhood of 
p and determine their weights w1,....wk. Determine the weighted mean value of the values stored under 
p1 .. .pk in level B 0 and store the result in memory level B, at the place belonging to p 

40 f. p:= successor of p 

g. is p the last pixel? 

h. f:= successor of f 

i. Is f the last color? 

j. Result : image memory level Bq contains the digitalized photo image data. Image memory levels B 1 B n 

45 contain image data for a complete, but preliminary (the "unfocussed") color or multispectral image 

Fig. 4 b 

a: Starting point : Image memory level B 0 contains the digitalized photo image data. The image memory levels 

so B 1 B n contain the data (calculated according to step 1) of the unfocussed color or multispectral image. 

b: p:=first pixel 

c: Read values Cq C n in the memory levels B 0 ,....Bn located at the position where p is to be found. Deter- 
mine color f belonging to p (see color determination). Calculate values C<\,...C n according to the formula 
Cj = Cj + C 0 - C f (additive image sharpening, or Ci = Cj • C 0 /C f for multiplicative image sharpening) and 
55 store Cj in the memory level Bj at the position where p is to be found, 
d: p: = successor of p 
e: Is p the last pixel? 

f: Result : image memory levels B n contain image data for a finished color or a multispectral image. 
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Claims 

1. Process for generating color images and/or multispectral images in which an optical arrangement (3) images a 
scene on a CCD image sensor (2), said sensor being covered by a mosaic filter with a periodic basic pattern (GM), 

5 characterized in that basic pattern (GM) consists of five different color filter elements and in that the process com- 
prises the following steps: 

storing values (wert[i][j][0]) of each individual pixel (py) of the CCD image sensor (2) in a dedicated level B 0 of 
an image memory (6), said values being measured by the CCD image sensor (2) and being transformed by a 

'0 following analog/digital converter (5) into digrtial values; 

determining an "unfocussed image" by calculating the neighboring pixel of color f for each color f and for each 
pixel p, reading their values from B 0 and combining them to arrive at a weighted average value; 
- storing values (wert[i][fl[f]) for all the different filter elements on the CCD image sensor (2) in dedicated image 
memory levels (B 1t B nF ); 

15 - calculating of final color values {C, ,C 2 C nF ) by means of additive image sharpening; and 

storing the final color values (C 1f C 2 , C nF ) in dedicated image memory levels (B 1 ^ B nF ). 

2. Process according to Claim 1 , characterized in that the final color values (C 1f C 2 ,C nF ) are calculated by multi- 
plicative image sharpening. 

20 

3. Process according to Claim 1 , characterized in that the illumination prevailing when photographing a scene can be 
transformed into an illumination of a different spectral composition. 

4. Process according to Claim 3, characterized in that when photographing a scene a white reference body with 
25 known spectral reflection properties is added to the scene to be photographed. 

5. Process according to Claim 3 or 4, characterized in that the data memory of processor (8) stores the data for dif- 
ferent illuminations, said data being transferable from the memory according to the chosen illumination, or charac- 
terized in that processor (8) uses the data obtained from the illuminated reference body to calculate the data 

6 necessary to make the change to a different spectral illumination and stores them in the memory of processor (8). 

6. Process according to Claim 1 , characterized in that the basic pattern (GM) used in the mosaic consists of a pen- 
taspectral pattern. 

35 7. Process according to Claim 6, characterized in that a distance from a starting pixel is defined in such a way that in 
the area defined by the chosen distance there are at least two pixels with the least frequently occurring color f of a 
filter element. 

8. Process according to Claim 7, characterized in that in the neighborhood of the starting pixel there are four to six 
40 pixels with the least frequently occurring fitter color. 

9. Process according to Claim 6, characterized in that the pixels at equal distance from the starting pixel are assigned 
to an equivalence class and in that these pixels have the same weighting factor w. 

45 1 0. Process according to Claim 9, characterized in that the equivalence classes and the weighting factors for a periodic 
basic pattern (GM) are stored in the memory of processor (8) in order to reduce the time to produce a final color 
image. 

System for generating color images and/or multispectral images by means of an optical arrangement (3) which 
images a scene on a CCD image sensor (2), said CCD image sensor (2) being covered by a mosaic filter with a 
periodic basic pattern (GM) and connected to a following analog/digital converter (5) which produces digital signals 
from all pixels p V} of the CCD image sensor (2), characterized in that the basic pattern (GM) is built up of at least five 
different filter elements. 

55 12. System according to Claim 11, characterized in that a digital image memory (6) with random access is provided, 

said memory having a number of image memory levels (B 1t B2 ,B nF ) which corresponds equally to the number 

of different filter elements of basic pattern (GM) and having in addition an image memory level B 0 which stores the 
digital signals of all pixels p, of CCD image sensor (2), said signals being generated by analog/digital converter (5). 
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13. System according to Claim 1 1 , characterized in that a processor (8) is used to control the system, organization and 
access of the image memory (6) and to evaluate the digital image data. 

14. System according to Claim 1 1 , characterized in that basic pattern (GM) is a pentaspectral pattern. 

15. System according to Claim 11, characterized in that the pentaspectral pattern is built up of three pixel rows and 
three pixel columns and in that pixels p 11f p 22 and p 33 are provided with a red filter element R, pixels p 13 . P21 and 
p 32 with a blue filter element B, pixel p 23 with a green filter element G, pixel p 12 with a green filter element G' and 
pixel p 31 with a green filter element G**, the maximum spectral transparency of G* being less than the maximum 

10 spectral transparency of G". 

16. System according to Claim 1 1 , characterized in that the pentaspectral Bayer pattern is built up of three pixel rows 
and three pixel columns, and in that pixels p 22 and D33 are provided with a red filter element R, pixels P21 and p 13 
with a blue filter element B, pixels p 12 and p 23 with a green filter element G. pixel p 32 with a green filter element G' 

15 and pixel p 1 1 with a green filter element G'\the maximum spectral transparency of G* being less than the maximum 
spectral transparency of G". 

17 System according to Claim 15 or 16, characterized in that the maximum spectral transparency of red filter element 
R is 595 nm, the maximum spectral transparency of blue filter element B is 445 nm. the maximum spectral trans- 

20 parency of green filter element G is 545 nm, the maximum spectral transparency of green filter element G' is 494 
nm and the maximum spectral transparency of green f ilter element G" is 570 nm. 

18 System according to Claim 12, characterized in that each individual level of image memory levels (B 1( B 2 ,....,B nF ) 
stores only data of a specific color value wert[i]UlPL and in that color values wertp][j][f] consist of the digital data of 

25 the color values of pixels Pij of the CCD image sensor (2), said pixels PiJ being provided with filter elements of the 
specific color f. and in that the digital values for pixels p 0 of the CCD image sensor (2) which are not provided with 
the filter element of the specific color f can be evaluated with the help of the processor (8). 

19. System according to Claim 18, characterized in that a distance from a starting pixel is defined in such a way that in 
the area defined by the chosen distance there are at least two pixels of the least frequently occurring color of a filter 
element. 

20. System according to Claim 19, characterized in that in the neighborhood of the starting pixel there are four to six 
pixels with the least frequently occurring filter color. 

21 . System according to Claim 20, characterized in that those pixels equidistant from the starting pixel are assigned to 
an equivalence class and in that these pixels have a constant weighting factor. 

22. System according to Claim 21. characterized in that in the case of a periodic basic pattern (GM) the equivalence 
40 classes ,and the weighting factors are stored in the memory of processor (8). 



30 



35 



45 



50 



55 



12 



MSDOCID: <EP 0804O37 A2_l_. > 



EP 0 804 037 A2 




SDOCID: <E P 0804Q37A2J_> 



13 



EP 0 804 037 A2 




IMSDOCID: <EP O804037A2J_> 



14 



EP 0 804 037 A2 




ISDOCID: <EP 0804037A2. L> 



15 



EP0 804 037 A2 



Blldauf nahme 



Dlgttallsleren 

Ergebnis; Per 
Bildspeicherbereich Bo enthdtt 
die AufnahmebikJdaten 




Nachfolger 
von p 



f :« 
Nachfolger 
von f 



Bestrmme ( siehe 
Nachbarschaftsfestiegung) die 

Pixel pv~pk der Farbe f, die zur 
Nachbarschaft von p gehftren, 
und deren Gewichte wi,...wk. 
Bilde den gewichteten Minelwert 
der zu pi,...pk in Bo gespeicherten 
Werte und speicbere das Ergebnis 
im Speicherbereich Bf an der zu p 
gehOrenden Stelie. 




Fig. 4a 



Ergebnis: 
Der Bildspeicherbereich Bo enthdft die 
digitalisierten Auf nahmebilddaten. Die 
Bildspeicherbereiche Bi, ...Bn enthatten 

Bilddaten fur ein vollstandiges, aber 
vorlfiuflges (das 'unscharfe') Farb- oder 
Multispektralbild. 
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Ausgangspunkt: 
Der Bildspeicherbereich Bo enthatt die 
digrtalsierten Aufnahmebilddaten. Die 
Bildspeicherbereiche Bi, ...Bn enthalten die 
gernaG Schritt 1 berechneten Daten des 
unscharfen Farb- oder Muftispektralbilds. 

p := erstes Pixel 



Nachfolger 
von p 



Lies <fie Werte Co,...Cn, die an der zu 

p gehOrenden Stelle in den 
Speicherbereichen Bo,..Bn stehen. 
Bestimme die Farbe f, diezu p gehOrt 
(siehe Farb best Immung). 
Berechne Werte Ci,...Cn gemaB 
Ci - Ci+Co-Cf 
(additive Blldschfirfung, Oder 

Ci = CrCo/Cf 
fOr multiplikative Bildschdrfung) 
und speichere Q im Speicherbereich 
Bi an der zu p gehGrenden Stelle. 




-neti 




Ergebnis; 

Die Bildspeicherbereiche Bi,...Bn 
enthaiten die Bilddaten fur ein 
fertlges Farb- Oder 
Multispcktralblld 



Fig. 4b 
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